Does willpower mindset really moderate the ego-depletion effect? A preregistered replication of Job, Dweck, and Walton (2010)

This article reports a preregistered study in which we attempted to replicate the results of an influential study on the ego-depletion effect reported by Job, Dweck, and Walton in 2010. The original Job et al. study (Study 1, N = 60) provided evidence that the ego-depletion effect—a performance decrease on a self-control task after performing another self-control task—occurs only for individuals who hold a belief that their willpower is limited. This moderation of the ego-depletion effect by one’s willpower mindset (limited vs. nonlimited) has been interpreted as evidence against a prevalent limited-resource account of self-control. Although this alternative account of the ego-depletion effect has become well-known, the statistical evidence of the original study was on shaky ground. We therefore conducted a preregistered replication of the original study with some methodological improvements. As in the original study, participants (N = 187) performed a self-control task (Stroop color-word interference task) after performing the control or depletion version of a letter cancelation task. Despite extensive analyses, we failed to replicate the original results: There was neither a significant main effect of ego depletion nor a significant moderation of this ego-depletion effect by individual differences in willpower mindset. Together with other recent failures to replicate the original moderation effect, our results cast doubts on the claim that an individual’s view of whether willpower is limited or not affects one’s susceptibility to the ego-depletion effect.

As is clear from Figure A, there was one extreme outlier in the control condition, whose error rate for Block 1 was over 60% (61.5%), well above the next highest error rates observed in either of the two conditions (control or depletion). As mentioned in the results section of the main article, this one extreme outlier had some noticeable impact on the statistical results, even though our sample size was almost three times as many as that for Job et al.'s original study.  Figure B (the outlier from Figure A is removed), the error data demonstrate a clear general tendency for the floor effect for both Blocks 1 and 2, because the error rates were low in our study. Arcsine-transformed accuracy data showed better distributional characteristics, but this transformation was not able to turn this skewed distribution into a normal distribution (for the results of the analyses based on arcsine-transformed Stroop accuracy data, see Table A   In contrast, the log RT data are more normally distributed for both blocks (Figure C), less skewed than the raw RT data we also analyzed (for the results of the fixed-effects and mixed-effects analyses based on raw RT data, see Table A and Table B).

Fixed-Effects and Mixed-Effects Modeling Results Based on Arcsine-Transformed Accuracy Data and Raw RT Data
This section provides the results of our supplementary (but preregistered) analyses based on arcsinetransformed accuracy data and raw RT data for both Stroop Block 1 and Block 2. (Note that Arcsine transformation has to be applied to the proportion correct measure, rather than the error rate or proportion incorrect measure).
We conducted the same fixed-effects and mixed-effects analyses reported in the main text, but with different dependent measures (see Table A for the results of the fixed-effects analyses and Table B for the mixed-effects analyses). The purpose of these additional analyses was to make sure that the results we report are robust to different data transformations. Indeed, the results based on the arcsinetransformed accuracy data and the raw RT data are consistent with those based on the raw error and log RT data reported in the main article (see Tables 4 and 5 in the main article for comparison). Note. Stroop effect (arcsine accuracy) was computed as the arcsine-transformed proportion of correct responses in incongruent trials minus the arcsine-transformed proportion of correct responses in congruent trials (Block 1) or asterisk trials (Block 2). B = unstandardized regression parameters. SE = standard errors for B. CI = 95% confidenceintervals for B estimates. Condition (between-subjects) was coded as -1 for the control condition and 1 for the depletion condition. Mindset = the beliefs about strenuous mental activity subscale score (between-subjects), which was mean-centered and standardized.
Note that Table B, which presents the results of the mixed-effects analyses, includes only the raw RT data (i.e., no analysis of the arcsine-transformed accuracy data). This is because the mixed-effects analyses of the error data we reported in Table 5 (the left panel) in the main article were logisticregression analyses focusing on the binary outcomes of the individual trials (correct or incorrect). Given that arcsine transformation must be applied to the proportion correct measure (not at the level of individual trials), it was not possible to conduct mixed-effects analyses for the arcsine-transformed accuracy data. This was the main reason for our decision to report the results of the raw error data in the main article, even though arcsine transformation helps spread out the scores in the high-accuracy range and generally improves the distributional characteristics of the accuracy data. Note. B = unstandardized regression parameters. OR = unstandardized odds ratios. SE = standard errors for OR and B. CI = 95% confidence-intervals for B estimates. Condition (between-subjects) was coded as -1 for the control condition and 1 for the depletion condition. Mindset = the beliefs about strenuous mental activity subscale score (betweensubjects), which was mean-centered and standardized. Trial type was coded as (-1 for congruent trials in Block 1 and neutral trials in Block 2 and 1 for incongruent trials in both Blocks 1 and 2). P-values are calculated based on Satterthwate's approximations.

Subblock of the Main Stroop Block 1 (the Replication Block)
As noted in the method section, Block 1 of the main Stroop task was intended to be the direct replication block. Because we increased the number of trials substantially (40 congruent and 40 incongruent trials in Block 1 of our study vs. 24 congruent and 24 incongruent trials in the original study by JDW2010), one could argue that, perhaps due to decay or dissipation of the depletion effect, the hypothesized effects of ego-depletion and its moderation by willpower mindset might be observed clearly if the analysis focused on the early portion of the Block 1 Stroop trials. Because our Block 1 Stroop trials were administered in two subblocks of 40 trials each, we conducted the same fixed-effects and mixed-effects analyses just for the initial subblock of the Block 1 Stroop trials (a total of 40 trials, which is comparable to the total number of Stroop trials administered in the original study [48]).
The relevant condition means (both the first and second subblocks of Block 1) are summarized in Table C below. As shown in the table, there was no evidence for the larger number of Stroop trials in this study contributed to the absence of the overall ego-depletion effect or its moderation by willpower mindset. The results of the fixed-effects analyses for the first subblock only are summarized in Table D, and the results of the corresponding mixed-effects analyses are summarized in Table E.  Note. B = regression parameters, not standardized with respect to the dependent variable. OR = unstandardized odds ratios. SE = standard errors for OR and B. CI = 95% confidence-intervals for B estimates. Condition (between-subjects) was coded as -1 for the control condition and 1 for the depletion condition. Mindset = the beliefs about strenuous mental activity subscale score (between-subjects), which was mean-centered and standardized. Trial type was coded as (-1 for the congruent trials and 1 for the incongruent trials). P-values for linear mixed effects models are calculated by lme4 package in R, based on Satterthwaite's approximations to degrees of freedom. P-values for binomial logistic mixed effects models are calculated using Wald's Z statistic as estimated by the lme4 package in R.

Results of Secondary Analyses with the Trait Self-Control Measure
In this section, we present the results of both fixed-effects and mixed-effects analyses we conducted for the trait self-control measure (we used the same models, but, instead of willpower mindset, we included trait self-control). Table F reports the results of the fixed-effects analyses, and Table G reports the results of mixed-effects analyses. The descriptive statistics for these questionnaire measures are provided in Table A above. As is clear from both Table F and G, there was no evidence for either the main effect of trait self-control on Stroop performance or the moderation of the ego-depletion effect by trait self-control. Note. B = unstandardized regression parameters. SE = standard errors for B. CI = 95% confidence-intervals for B estimates. Condition (between-subjects) was coded as -1 for the control condition and 1 for the depletion condition. TSC = the trait self-control scale score (between-subjects), which was mean-centered and standardized. Note. B = Unstandardized regression parameters. OR = Unstandardized odds ratios. SE = Standard errors for OR and B. CI = 95% confidence-intervals for B estimates. Condition (between-subjects) was coded as -1 for the control condition and +1 for the depletion condition. TSC = the trait self-control scale score (between-subjects), which was mean-centered and standardized. Trial type was coded as (-1 for the congruent trials in Block 1 and the neutral (asterisk) trials in Block 2 and 1 for the incongruent trials for both Blocks 1 and 2). P values for binomial logistic mixed-effects models (for the error data) were calculated using Wald's Z statistic as estimated by the lme4 package in R. P values for linear mixed-effects models (for the log RT data) are calculated by lme4 package in R, based on Satterthwaite's approximations to degrees of freedom.